Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset

Scene motion, multiple reflections, and sensor noise introduce artifacts in the depth reconstruction performed by time-of-flight cameras. We propose a two-stage, deep-learning approach to address all of these sources of artifacts simultaneously. We also introduce FLAT, a synthetic dataset of 2000 ToF measurements that capture all of these nonidealities, and allows to simulate different camera hardware. Using the Kinect 2 camera as a baseline, we show improved reconstruction errors over state-of-the-art methods, on both simulated and real data.Comment: ECCV 201

Femto-Photography: Capturing Light in Motion

We show a technique to capture ultrafast movies of light in motion and synthesize physically valid visualizations. The effective exposure time for each frame is under two picoseconds (ps). Capturing a 2D video with this time resolution is highly challenging, given the extermely low SNR associated with a picosecond exposure time, as well as the absence of 2D cameras that can provide such a shutter speed. We re-purpose modern imaging hardware to record an ensemble average of repeatable events that are synchronized to a streak tube, and we introduce reconstruction methods to visualize propagation of light pulses through macroscopic scenes. Capturing two-dimensional movies with picosecond resolution, we observe many interesting and complex light transport effects, including multibounce scattering, delayed mirror reflections, and subsurface scattering. We notice that the time instances recorded by the camera, i.e. “camera time” is different from the the time of the events as they happen locally at the scene location, i.e. world time. We introduce a notion of time warp between the two space time coordinate systems, and rewarp the space-time movie for a different perspective

A Biophysically-Based Model of the Optical Properties of Skin Aging

This paper presents a time-varying, multi-layered biophysically-based model of the optical properties of human skin, suitable for simulating appearance changes due to aging. We have identified the key aspects that cause such changes, both in terms of the structure of skin and its chromophore concentrations, and rely on the extensive medical and optical tissue literature for accurate data. Our model can be expressed in terms of biophysical parameters, optical parameters commonly used in graphics and rendering (such as spectral absorption and scattering coefficients), or more intuitively with higher-level parameters such as age, gender, skin care or skin type. It can be used with any rendering algorithm that uses diffusion profiles, and it allows to automatically simulate different types of skin at different stages of aging, avoiding the need for artistic input or costly capture processes

Practical Multiple Scattering for Rough Surfaces

Microfacet theory concisely models light transport over rough surfaces. Specular reflection is the result of single mirror reflections on each facet, while exact computation of multiple scattering is either neglected, or modeled using costly importance sampling techniques. Practical but accurate simulation of multiple scattering in microfacet theory thus remains an open challenge. In this work, we revisit the traditional V-groove cavity model and derive an analytical, cost-effective solution for multiple scattering in rough surfaces. Our kaleidoscopic model is made up of both real and virtual V-grooves, and allows us to calculate higher-order scattering in the microfacets in an analytical fashion. We then extend our model to include nonsymmetric grooves, allowing for additional degrees of freedom on the surface geometry, improving multiple reflections at grazing angles with backward compatibility to traditional normal distribution functions. We validate the accuracy of our model against ground-truth Monte Carlo simulations, and demonstrate its flexibility on anisotropic and textured materials. Our model is analytical, does not introduce significant cost and variance, can be seamless integrated in any rendering engine, preserves reciprocity and energy conservation, and is suitable for bidirectional methods

Progressive Transient Photon Beams

In this work, we introduce a novel algorithm for transient rendering in participating media. Our method is consistent, robust and is able to generate animations of time-resolved light transport featuring complex caustic light paths in media. We base our method on the observation that the spatial continuity provides an increased coverage of the temporal domain, and generalize photon beams to transient-state. We extend stead-state photon beam radiance estimates to include the temporal domain. Then, we develop a progressive variant of our approach which provably converges to the correct solution using finite memory by averaging independent realizations of the estimates with progressively reduced kernel bandwidths. We derive the optimal convergence rates accounting for space and time kernels, and demonstrate our method against previous consistent transient rendering methods for participating media

Reference normal values and design of a vision screening for 4 to 5 years old preschoolers

A vision screening program for preschool children of 4-5 years old was designed and analyzed. Information of the prevalence of ocular conditions among preschool children was obtained. The vision health of a group of 127 children was evaluated by a comprehensive examination in their own school. If a child failed one or more screening tests, he was referred to the ophthalmologist. Of the children screened in this study, 61% passed distance visual acuity and retinoscopy tests, 17% were referred to the ophthalmologist and 22% will be annually monitoring. Values of monocular/binocular acuity worse than 0.5/0.6 are too poor for 4 years old children, whereas these limits increase up to 0.6/0.8 for 5 years old children. In conclusion, the prevalence of undetected vision problems in preschool children has been clearly demonstrated. Vision screening programs in schools are highly recommended. Nevertheless, coordination among professionals conducting screening, school personnel and parents are needed to reach high levels of success. The results of this study validate an easy and fast battery of tests. The vision screening has been highly reliable because reference normal values have been defined by analyzing statistically the results of these tests. Se diseñó y analizó un screening visual en preescolares de 4-5 años de edad. Se obtuvo información de la prevalencia de las condiciones oculares entre estos preescolares. Se evaluó la salud visual de un grupo de 127 niños con un examen completo en su propia escuela. Si el niño fallaba uno o más test era referido al oftalmólogo. De los niños revisados en el estudio, el 61% superó los test de agudeza visual y retinoscopía, al 17% se le refirió al oftalmólogo y el 22% será controlado anualmente. Valores de agudeza monocular/binocular peores de 0.5/0.6 son demasiado bajos para niños de 4 años de edad, mientras que estos límites se incrementan a 0.6/0.8 para niños de 5 años. En conclusión, la prevalencia de problemas de visión sin detectar en niños preescolares ha sido claramente demostrada. Sin embargo, es necesaria la coordinación entre profesionales que realicen el screening, personal del colegio y padres para alcanzar niveles altos de éxito. Los resultados de este estudio validan una batería rápida de test. El screening visual ha sido altamente fiable porque los valores normales de referencia han sido definidos analizando estadísticamente los resultados de estos tests

Differentiable Transient Rendering

Recent differentiable rendering techniques have become key tools to tackle many inverse problems in graphics and vision. Existing models, however, assume steady-state light transport, i.e., infinite speed of light. While this is a safe assumption for many applications, recent advances in ultrafast imaging leverage the wealth of information that can be extracted from the exact time of flight of light. In this context, physically-based transient rendering allows to efficiently simulate and analyze light transport considering that the speed of light is indeed finite. In this paper, we introduce a novel differentiable transient rendering framework, to help bring the potential of differentiable approaches into the transient regime. To differentiate the transient path integral we need to take into account that scattering events at path vertices are no longer independent; instead, tracking the time of flight of light requires treating such scattering events at path vertices jointly as a multidimensional, evolving manifold. We thus turn to the generalized transport theorem, and introduce a novel correlated importance term, which links the time-integrated contribution of a path to its light throughput, and allows us to handle discontinuities in the light and sensor functions. Last, we present results in several challenging scenarios where the time of flight of light plays an important role such as optimizing indices of refraction, non-line-of-sight tracking with nonplanar relay walls, and non-line-of-sight tracking around two corners

Non-line-of-sight transient rendering

The capture and analysis of light in flight, or light in transient state, has enabled applications such as range imaging, reflectance estimation and especially non-line-of-sight (NLOS) imaging. For this last case, hidden geometry can be reconstructed using time-resolved measurements of indirect diffuse light emitted by a laser. Transient rendering is a key tool for developing such new applications, significantly more challenging than its steady-state counterpart. In this work, we introduce a set of simple yet effective subpath sampling techniques targeting transient light transport simulation in occluded scenes. We analyze the usual capture setups of NLOS scenes, where both the camera and light sources are focused on particular points in the scene. Also, the hidden geometry can be difficult to sample using conventional techniques. We leverage that configuration to reduce the integration path space. We implement our techniques in a modified version of Mitsuba 2 adapted for transient light transport, allowing us to support parallelization, polarization, and differentiable rendering. © 2022 The Author(s

Femto-photography: capturing and visualizing the propagation of light

We present femto-photography, a novel imaging technique to capture and visualize the propagation of light. With an effective exposure time of 1.85 picoseconds (ps) per frame, we reconstruct movies of ultrafast events at an equivalent resolution of about one half trillion frames per second. Because cameras with this shutter speed do not exist, we re-purpose modern imaging hardware to record an ensemble average of repeatable events that are synchronized to a streak sensor, in which the time of arrival of light from the scene is coded in one of the sensor's spatial dimensions. We introduce reconstruction methods that allow us to visualize the propagation of femtosecond light pulses through macroscopic scenes; at such fast resolution, we must consider the notion of time-unwarping between the camera's and the world's space-time coordinate systems to take into account effects associated with the finite speed of light. We apply our femto-photography technique to visualizations of very different scenes, which allow us to observe the rich dynamics of time-resolved light transport effects, including scattering, specular reflections, diffuse interreflections, diffraction, caustics, and subsurface scattering. Our work has potential applications in artistic, educational, and scientific visualizations; industrial imaging to analyze material properties; and medical imaging to reconstruct subsurface elements. In addition, our time-resolved technique may motivate new forms of computational photography.MIT Media Lab ConsortiumLincoln LaboratoryMassachusetts Institute of Technology. Institute for Soldier NanotechnologiesAlfred P. Sloan Foundation (Research Fellowship)United States. Defense Advanced Research Projects Agency (Young Faculty Award

Adaptive polarization-difference transient imaging for depth estimation in scattering media

Introducing polarization into transient imaging improves depth estimation in participating media, by discriminating reflective from scattered light transport and calculating depth from the former component only. Previous works have leveraged this approach under the assumption of uniform polarization properties. However, the orientation and intensity of polarization inside scattering media is nonuniform, both in the spatial and temporal domains. As a result of this simplifying assumption, the accuracy of the estimated depth worsens significantly as the optical thickness of the medium increases. In this Letter, we introduce a novel adaptive polarization-difference method for transient imaging, taking into account the nonuniform nature of polarization in scattering media. Our results demonstrate a superior performance for impulse-based transient imaging over previous unpolarized or uniform approaches